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ABSTBACT 

An evaluation is provided of ten fundamentally 
different techniques applicable for choosing instructional media for 
proposed Navy training programs* The evaluative method used employed 
six training system designers, each of whom applied the chosen 
techniques to a sample of seven representative training tasks* The 
results of these applications were then examined by a panel of 
experts who judged the appropriateness of the media chosen to the 
task's training requirements* This served as the basis for ranking 
the techniques in terms of their usefulness in the design of Navy 
training programs* The rating for each of the three top ranked 
techniques were essentially similar and no clear superiority was 
established* Furthermore, none of the techniques were found adequate 
for direct application to Navy training programs* However, one of the 
three, the Training Analysis and Evaluation Group (TAEG) technique, 
was selected as the logical choice for further development and 
guidelines for its modification are presented* (Author) 
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AN EVALUATION OF TEN TECHNIQUES 
FOR CHOOSING INSTRUCTIONAL MEDIA 

ABSTRACT 

This report provides an evaluation of 10 fundamentally different 
techniques applicable for choosing instructional media for proposed Navy 
training programs. 

The method of evaluation used involved six training system designers 
each of whom applied the chosen techniques to a sample of seven representa- 
tive Navy training tasks. The results of this application were then ex- 
amined by a panel of experts who judged the appropriateness of the media 
chosen to the task's training requirements. This served as the basis for 
ranking the techniques in terms of their usefulness in the design of Navy 
training programs. 

The ratings for each of the top three ranked techniques were 
essentially similar in value. Based on these ratings, no clear cut superior- 
ity could be ascribed to any of these three techniques. Further, none of 
the techniques was found adequate for direct application to the Navy. 
However, one of the three, the TAEG technique, was selected as the logical 
choice for further development. Guidelines for its modification are 
presented . 
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SECTION I 
INTRODUCTION 

Choosing a cost-effective mix of instructional media for a military 
training program has proved to be a difficult task. Existing formal 
procedures for media selection have been essentially ineffective. For one 
thing, the procedures are inexact; the selection criteria are too general 
for identifying specific media for specific training. The level of detail 
is also at issue. Some procedures are too simple to handle the selection 
problem; most are too complicated and cannot be adapted easily to a range 
of military needs. Various factors contribute substantially to the com- 
plexities of the media selection effort. The more prominent among these 
are outlined below. 

a. An intrinsic difficulty in optimizing the media mix exists due 
to the nature of military jobs. Most people agree that the selection of 
training supports is most effective when unique media attributes can be 
associated with specific training objectives or subject matter activities. 
However, military operations demand a variety of skills and knowledges in 
job performance. Numerous activities are conducted simultaneously or inter- 
actively such that it is difficult to identify those task characteristics 
relevant to media selection. The frequent 'requirement for manual control 
activities throughout job performance further compounds the difficulty in 
analyzing job task requirements. Thus, job performance may involve relative- 
ly straightforward procedural and discrete acts (positioning controls, com- 
municating), perceptual-discriminative acts (identifying, monitoring, antic- 
ipatory responses), and perceptual-motor acts (graded response in continuous 
interaction with stimulus changes). Emphasis is placed on the integration 
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of responses, coordination and timing, time sharing, decision-making and 
judgment. These activities are not easily partitioned and grouped for train 
ing. The inability to conveniently identify discrete beginnings and endings 
for behavioral processes hampers the selection of media appropriate to given 
job requirements. 

b. It is difficult to correlate the employment of a specific in- 
structional medium with success in task learning. While a medium provides 
known capabilities that can be exploited in^ training, it is the quality of 
the courseware and the utilization procedures that are major determiners 
of the success of the training effort. 

c. The proliferation of instructional hardware during the past 
decade has resulted in a wide range of new equipments, yet unbiased in- 
formation is lacking for evaluating the claims made for these innovations. 

d. Cost is a factor in choosing media. Not only do competitive 
training media vary in initial and operating costs, but each has unique 
costs and requirements for facilities, personnel, and supplies. The time 
required to achieve the objectives of training also varies among media. 
Therefore, the cost of achieving instructional goals varies significantly 
from medium to medium. The training system designer must not only specify 
a training system capable of accomplishing the instructional goals but must 
also consider these decisions in terms of cost and lead time factors. 

e. While various media selection techniques have been described 
and demonstrated, there is little evidence any of these have been experi- 
mentally evaluated or compared. No formal attempts have been made to 
establish the relative reliability and validity of these techniques for 
selecting instructional media for proposed training systems. The training 
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system designer is not provided the necessary basis for determining which 

technique(s) is suitable for his needs, 

PURPOSE 

The present report provides an organization of information pertaining 
to the utility of media selection techniques for training system design. 
It provides an evaluation of prominent media selection techniques and a 
comparison of the relative value of these techniques for the Training Analy- 
sis and Evaluation Croup's (TAEG's)^ use with Navy training. The specific 
objectives of the study are: 

a. Review and place in perspective a representative sample of the 
published formal media selection techniques . The techniques chosen reflect 
fundamentally different approaches. As a condition for selection, each 
technique was required to produce at least two media options for each learn- 
ing event. The techniques ranged from simple to complex in terms of the 
number of variables accounted for, the precision of the definition of terms, 
and the operations performed (factors integrated). They differed also in 
the way the analyses were performed, varying from wholly manual procedures 
to combinations of manual and automated modes. 

Our review of these techniques provides a description of each 
approach, an examination of its logic, and an assessment of the practicality 
of the procedures for use in training system design. 

b , Evaluate the usefulness of the techniques sampled and de - 
termine a rank-order priority of usefulness . Ten techniques were examined 

\he Training Analysis and Evaluation Group of the Naval Training Equipment 
Center, Orlando, Florida, is a multi-disciplinary group tasked by the 
Chief of Naval Education and Training. One of its functions is to prepare 
training system specifications for proposed Navy training systems. 
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in terras of their power to aid Navy training system designers in optimizing 
media choices, A description of each technique is provided in appendix A. 

One of the techniques chosen for evaluation was the recently de- 
veloped Training Effectiveness and Cost Effecti veness Prediction (TECEP ) 
Model developed by the TAEG (TAEG Report No. 1, 1972). Another technique 
chosen required the training system designer to choose media using only his 
intuitive judgment and experience, without the aid of formal media selection 
logic. This "Non-System" technique, which Velies cn the expertise of the 
individual, has been conventionally employed by training system designers. 
This intuitive approach was added to the nine formal techniques to serve 
as a control to determine the extent to which a given formal technique 
would influence the selection of media for defined training segments. The 
evaluation of the chosen techniques was undertaken to measure the usefulness 
of these techniques in dealing with a broad range of Navy training tasks. 
The selection of a representative sample of tasks for use in the evaluation 
was based on two sources: (1) 19 Categories of Navy tasks defined by 
Willis and Peterson (1961) and (2) the ranking of these task categories 
by their frequency of occurrence in a sample of critical Navy jobs by 
Bernstein and Gonzales (1971). From these categories and priorities, 
seven task categories were assembled which represent, in our opinion, the 
most common activities performed across a range of Navy jobs. Training 
objectives were written for each of these seven task categories. 

c. Provide design recommendations for improving the most 
promising media selection approach in our sample of 10 techniques for 
TAEG operations . 
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ORGANIZATION OF THE REPORT 

In order to organize and discuss the analytic procedures and the re- 
sults of the evaluation, three major sections are presented in addition to 
this introductory section. 

Section II describes the specific behavioral objectives, the media 
selection techniques, and the experimental procedures used in making the 
comparisons. It ai30 presents the qualifications of the training system 
designers who employed the various techniques and of the experts who rated 
the output of these techniques. 

Section III presents the results of the analyses and provides the 
opinions of the training system designers about the acceptability and 
utility of each of the 10 systems and of the time required to learn and 
thpn use each technique. Also, analyses are provided which indicate the 
significant relations between specific media StiJ^tTtion techniques and 
specific behavioral objectives. 

Section IV presents a statement of the usefulness of the media 
selection techniques and identifies the common characteristics found in 
the more useful approaches. The most promising technique for the TAEG's 
use is identified together with a set of criteria for improving this 
technique, • 

In addition, two appendices are provided. Appendix A contains a 
detailed description of the media selection techniques selected for 
study; appendix B contains the instructions used in briefing the training 
system designers prior to their use of the 10 media selection techniques. 
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SECTION II 
APPROACH 

The method of evaluation involved the direct application of each of 
the chosen media selection techniques to a sample of representative Navy 
training tasks. The results of this application were then examined by a 
panel of experts who judged the appropriateness of the media chosen to the 
task*s training requirements. This served as the basis for ranking the 
techniques in terms of their usefulness to the TAEG*s purpose. 

The procedures employed required six training system designers to 
learn and then use 10 different media selection techniques. Each designer 
independently applied each of the techniques to seven standard training 
tasks. This produced a total of 420 sets of media choices. Examining this 
pool of media choices, two experts in training system design independently 
rated each media choice for its ability to meet the training objectives. 
The ratings of all media choices for each of the 10 techniques were then 
combined and the relative utility of each technique for media selection was 
determined. 

The materials developed in support of the study, the personnel in- 
volved in the study, and the procedures employed in the evaluation are 
described next. , 
MATERIALS 

The following materials were prepared for use in the evaluation: 
a. Training Objectives . Based on the Bernstein and Gonzales 
(1971) ranking of common Navy task categories, the top seven categories 
were selected and modified for use in this study. In the order of their 
significance, these categories are: 
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(1) Recalling facts and principles 

(2) Recalling procedures 

(3) Non-verbal identification (classification) 

(4) Detection (non-verbal uses) 

(5) Using principles, interpreting, inferring 

(6) Decision-making 

(7) Static reaction and continuous movement 

For each of these task categories, a sample training objective was prepared. 
A list of these objectives is shown in table 1. 

b. Media Selection Techniques . Ten media selection techniques 
were chosen for this evaluation. Five criteria were established for deter- 
mining the media selection techniqueis to be evaluated: 

(1) Techniques that had been used in major training system 
design projects vTere considered as prime candidates for the study. 

(2) The output of a given technique had to provide two or more 
useful media alternatives for a given training objective(s), not just a 
single best choice. 

(3) Representative techniques involving military and civilian 
application were sought. 

(4) Fully developed procedures, rather than just descriptions 
of a process used in a particular applicationj were sought. However^ where 
the description was adequate, a given technique was considered a candidate. 

(5) The techniques eventually selected should represent signi- 
ficantly different approaches to media selection. 

Based on these criteria, an exclusive sample of 10 media selection 
techniques was chosen. 
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TABLE 1. SAMPLE TRAINING OBJECTIVES REPRESENTING HIGH PRIORITY 
NAVY TASK CATEGORIES 



TRAINING OBJECTIVES 

1. Upon request, student will write 
Ohms Law and define each symbol 
correctly. 



2. Without coaching, a student will 
properly activate and calibrate 
an Ohm meter, and measure the 
resistance in an unmarked 
resistor within the range of 
2000 to 100,000 ohms with a pre- 
cision of + 5 percent. 

! 3. When presented with 10 pulse 

analyzer photographs of differ- 
ent search radars, four of which 
are long range air search radars, 
the student will . correctly iden- 
tify all the long range air 

: search radars within 10 seconds. 

: 4. When presented with random one- 
i second pulses of a 5000 Hertz 

tone emerging slowly from a white 
noise background the student will 
detect the tone before it reaches 
a signal-to-noise differential 
of 15 DB, 80 percent of the time. 



ENTERING BEHAVIORS 

Student can write English with ade- 
quate skill but is not familiar 
with the formula for Ohms Law and 
cannot provide the technical defi- 
nitions for the factors in the law. 
He is not familiar with the formal 
concepts "voltage," "amperage," or 
"resistance." 

Student can write and define Ohms 
Law and can solve the Ohms Law 
equation for resistance when the 
other variables are given. He 
cannot set up the equipment nor 
collect the necessary data from a 
live circuit. 

Student can identify and measure 
pulse width, pulse length, and other 
elements of a radar signature and 
can recall the concepts basic to any 
signature configuration. He cannot 
identify specific classes of sig- 
natures* 

Student can identify the obvious 
tone with a signal-to-noise differ- 
ential of 30 DB. He is physiolog- 
ically capable of hearing the tone 
at a differential of 15 DB but 
cannot as yet detect it in the back- 
ground noise. 
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TABLE 1. SAMPLE TRAINING OBJECTIVES REPRESENTING HIGH PRIORITY 
NAVY TASK CATEGORIES (CONT'D) 



7. 



TRAINING OBJECTIVES 

Wlien presented with a paper and 
pencil exercise concerning the 
bringing of a Captain's gig 
alongside a pier in a two-knot 
cross current , the student cox- 
swain shall diagram his approach 
to the pier and identify the 
changes in power setting and 
rudder he would make to correctly 
compensate for the cross current 
and to account for the power and 
turning characteristics of the 
boat . 

Given an EW mission objective to 
locate ard analyze the long 
range air search radars on a 
specified island using a speci- 
fied platform and equipment set, 
plan an EW mission capable of 
obtaining the required informa- 
tion without being detected by 
the long range radars. 

While manning the ship control 
station on an SSN in an in-trim 
condition submerged at 80 feet 
with 10 knots, a ship control 
team shall execute a change in 
depth of 100 feet in two minutes 
and stabilize at ordered depth 
for two minutes with excursions 
of no more than five feet. 



ENTERING BEHAVIORS 

The student can already recall the 
various individual principles 
required, but he has not attempted 
to apply one or more of these 
principles to realistic problems. 



Student can operate the EW systems 
required in the proposed mission and 
can identify the expected target 
signals. He has the required back- 
ground in EW system management, but 
he has not seen mission plan formats 
nor has he prepared or evaluated 
mission plans. 



The team consists of an experienced 
diving officer and Ballast Control 
Panel Operator with trainee bow and 
stern plane operators that have not 
reached the required proficiency 
level. Excursions are in excess of 
five feet. 
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The techniques chosen are listed in table 2 with brief statements 
concerning the purpose of their development and where applied* More 
detailed descriptive information on each technique is given in appendix A« 

As indicated earlier, one of the techniques, the Non-System approach, 
was added to represent the conventional approach to media selection. In 
this case the training system designer relies on his skill, past experience, 
and personal preference in selecting the media for a program without resort- 
ing to any formal procedure* This approach was included as a control to 
determine to what extent the use of a formal technique makes a difference in 
the choice of media. 

Another of the techniques, the TAEG approach, was included to determine 
the usefulness of this locally developed prototype system in comparison to 
other formal systems. 

The nine formal techniques differed substantially in the number of 

and kind of variables considered important lo the media selection process. 

A total of 14 variables were identified, applicable to one or more of the 

techniques* Table 3 lists the 14 variables and identifies (by an »'X'* in a 

cell) which variables were involved in a given media selection technique* 

In addition, the last column of table 3 shows the total number of media 

« 

candidates available in each technique* The variation among the chosen 
techniques is conside^rable, ranging from five media alternatives in one 
technique to 31 alternatives in another* In one case the media candidates 
were not listed, the choice(s) being left to the judgment of the training 
specialist. While some of the media selection techniques used similar 
variables, each technique employed a different combination of variables. 
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TABLE 2. MEDIA SELECTION TECHNIQLES USED IN THE EVALUATION 



No. Author Sponsoring Organization Purpose, of Development Notes 




0 "Non-System" Traditional approach. The 

training system designer 
used hlB ovn ejcperlence and 
preferences without the aii 
of a formal media selec*- 
tlon technique. 


1 "AF Manual Department of Air Force; part of a guide for applying the Technique was developed by 
50-2" Air Training Command Air Force systems approach to the Dr. Wm H. Allen and was 
(31 Dec 1970) development o:l education and train- originally presented In a 

ing programs; to be used by all paper "Research in Instruc- 
nersonnel who \ilan. develoD. ^^nnA1 MpjUa jmH A^-^ rr^iir>A~ 
approve, administer, or manage Air tlon" which was published 
Force instruction and Its support- In Final Report of the Uses 
ing services. of Newer Media in Art Edu- 
cation Prolect. 


2 Bretz The Rand Corporation Part of a large Rand investigation Concerns telecommunlca- 
(Feb 1971) (for Headquarters, USAF) of systems for Air Force Education tlon and recorded media. 

and Training; to bv. used In design- 
ing technical training programs to 
identify appropriate communication 
media. 




"a^bca a x\ppxj.t:u r0y1.11uxug1.uax Lo rcuuouacnu a nsw uevAce ano oaseo on J* uulliord s 
Fedennan Services, Inc. (for the classroom aids to be used in a "Structure of Intellect" 
<Jun 1970) Naval Training Device proposed training progr&Ti for model. 
Center) Che Tactical Coordlnatorfi (TACCO) 

in ASW aircraft. 


4 TAEG (1972) Training Analysis & For use within TAEG In choosing Partially based on Willis 

Evaluation Group of the cost-effective media for proposed & Peterson, Deriving Train- 
Naval Training Equip- training systems to be used in ing Device Implications 
roent Center preparing training oystem sped-' from Learning Theory 

fl cations. Principles, Vol. I, 


5 Briggs(1970) American Institutes for For use In a handbook of resource Uses Gagne*s 8 types of 
Research and Florida materials and methods for teaching learning and related 
State University the systems approach to the design conditions of learning, 

of instruction. Oriented to the 

teaching of academic subjects in 

classrooms. 


6 Armstrong, Bunker Ramo (for Air A component of the system? Media weights within system 
al. Force Human Resources approach to training (SAT) to be were des'gned for A-7D 
(Aug 1971) Laboratory) used in designing training for training, and cannot be 

the AK A-7D aircraft. considered «ippropriate for 

other tasks » 


7 Rhode, Westinghouse Learning Initial study In the design of an Not intended to be a media 
et al. Corp, (for Air Force advanced multi-media Instructional selection technique, but 
(May 1970) Human Resources system; to compare media In terms provides a wealth of In for - 
Laboratory) of function, flexibility and cost; mation to support ^he 

for use with a broad range of Air training system des>^.(;ner 
Force training tasks. In choosing media. 




8 Walker Martin Company A general method for use with a Based on the rating of 

(1967) broad range of training tasks, training techniques by ex-* 

perlenced training person- 
nels Basic approach being 
used by Army at HumRRO, 
Ft Knox, KY. 


9 Roucher, Grumman Aerospace For use in selecting media for the Contains an extensive com- 
et al, Corp (for Navy) Navy F-14 training system and puter data bank of media, 
(Oct 1971) other Navy training tasks, media characteristics, and 

/ costs, with automatic sort- 
ing to list all media with 
t a given set of character- 
istics. 



NOTE; Sources of these techniques are listed in bibliography and 
additional descriptive information is located in appendix A. 
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This indicates clearly that fundamentally different techniques were 
compared. 

Separate folders were prepared for each of the media selection 
techniques. Each folder contained: (1) a copy of the original document 
describing the technique, and (2) step- by- step procedures developed by 
TAEG from the original documentation for carrying out the intent of the 
technique. Identification of the author(s) and the sponsoring organizations 
was removed from the folder to safeguard against influencing the evaluators. 

c. Training System Designer Rating Scales . Three scales were pre- 
pared for the training systems designers to use in recording their overall 
evaluation of the utility of each of the competitive media selection 
techniques. The first scale concerns the detail in the media prescription 
and .ranges from "broad media categories" on one end to "specific media 
categories with detailed specification data" on the other end. The intent 
here was to document the degree of detail to be found in the description of 
a proposed type of media. The second scale concerns the level of confidence 
the training system designer has in the validity of the media selected and, 
therefore, the media selection technique. It ranges from "no confidence'* 

to "high confidence,*' The third scale concerns the training system designer's 
estimate of the suitability of the technique for use in TAEG projects. This 
is a general measure and includes ease of use, time to use, as well as the 
previously mentioned factors of validity and detail of media prescription. 
Scale values range from "under no circumstances should TAEG use this system" 
to "an ideal system for general use in TAEG.." Figure I depicts these scales. 

d. Expert Rating Scale . An additional rating scale was developed 
for use by the expert training system designers in rating the media choices 
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made employing the various media selection techniques. This scale ranged 
from "mostly poor media choices" to "comprehensive set of media alternatives." 
Figure 2 depicts the full scale. 
PERSONNEL 

Two groups of training specialists were Involved in the evaluation. 
One group, designated as training system designers, employed each of the 
chosen media selection techniques in deriving an optimum media mix to 
achieve the training objectives for each of* the seven designated tasks. 
The other group, designated as experts in training system design, evaluated 
and* then formally rated the media choices selected by the training system 
designers. 

TRAINING SYSTEM DESIGNERS . Six professional members of the TAEG were 
assigned as training system designers to employ the chosen media selection 
techniques. Three were psychologists, two were educators, and one was an 
operations research analyst. Er»cept for one of the education specialists, 
all had previously worked as members of training system design teams. None 
had previously used any of the chosen media selection techniques. 
EXPERT JUDGES . Two experts in training system design were selected to rate 
the complete set of media choices made by the training system designers. 
One was Dr. Alfred F. Smode, formerly the executive scientist for Dunlap 
and Associates, Inc., Darien, Connecticut, and presently the senior scientist 
of TAEG. Dr. Smode has more than 20 years experience in Human Factors con- 
sulting and in training research and application. The second expert was 
John D. Armstrong, who at the time was the senior education specialist with 
TAEG. Mr. Armstrong has 22 years experience as an advisor to Navy schools 
on the use of training equipment and has recently completed a Navy-wide 
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survey on the products and services required to support Navy training. 
PROCEDURES 

USE OF MEDIA SELECTION TECHNIQUES . The training system designers were 
briefed on the general procedures to be followed. The scenario for this 
briefing is provided in appendix B. Following the briefing, the training 
system designers first employed the Non-System approach; then each of the 
other nine media selection techniques was utilized. The order in which 
these techniques were assigned to the training system designers was varied 
to Insure that no two individuals wuld apply the techniques in a similar 
order. 

Before applying a given technique, the training system designer studied 
the folder until he understood the process and recorded the time required 
to learn the technique. He then proceeded to use the technique to choose 
media for each of the training objectives, starting with objective number 
one. All the media types identified as useful by the media selection 
techniques were recorded. After using a media selection technique with all 
seven objectives, the training system designer recorded the time required 
to apply the given technique and chen rated the technique according to the 
three scales shown in figure 1. 

RATING OF MEDIA CHOICES BY EXPERTS . The experts independently reviewed the 
training objectives and used the scale shown in figure 2 to rate each of 
the media choices made by the training system designers. The ratings 
represented the experts' estimate of the usefulness of the media choices 
in enabling trainees to meet the specific training objectives. The rating 
values were assigned without knowledge of which training system designer 
had made the media choices or which media selection technique had been used. 
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SECTION III 
RESULTS 

Three types of data were obtained from the trials of the media se- 
lection techniques. First, the training system designers generated 420 
sets of media choices which were rated by two experts in terms of their 
usefulness in achieving specific training objectives. Second, each train- 
ing system designer rated each media selection technique in terms of his 
estimate of its usefulness in the design of Navy training programs. And 
third, each training system designer recorded the time used to learn each 
technique and the time required to apply each technique to seven training 
objectives. 

Table 4 presents the experts* rating of the 420 sets of media choices. 
It also presents the rank order correlations (Rho) between the ratings 
of the two experts. The ratings made by the experts varied from a Rho of 
.97 on training objective 7 to a Rho of -,60 on training objective 5. 
A relatively high correlation was achieved between the scores assigned by 
the two experts on four of the objectives. On three of the objectives there 
was one low positive and two negative correlations. In these instances, 
the experts disagreed on what constituted a useful solution. While this 
variation can in part be attributable to ambiguity in the training objectives, 
it also highlights the feature of the artificiality of the rating task. The 
expert ratings were made with difficulty in that the training objectives 
were described out of context to any larger training effort and many of 
the media choices were described with insufficient detail to insure that 
the experts were considering and rating r.he same forms of media. Part of 
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the variance is also attributable to the fact that the experts disagreed 
on the optimal solution. 

Because of the marked differences between the two experts in the 
rating of the media choices for three of the training objectives, the 
ratings of the experts were combined for use in ranking the media selection 
techniques. Table 5 summarizes the ratings made by the experts. The sum 
and mean rating made by each and their combined ratings are shown. The 
combined scores provide a rank order of the various media selection 
techniques in terms of their usefulness to the Navy as broad purpose media 
selection techniques. 

The highest ranked technique was the TAEG technique with a mean 
rating of 3,60, This was followed closely by the Non-System technique with 
a mean rating of 3,55; then Briggs with a mean rating of 3,51; Armstrong 
with a mean rating of 3,44; and AF Manual 50-2 with a mean rating of 3.18, 

Table 6 shows the summary of an overall 2- factorial analysis of variance 
(ANOVA) performed on the data in table 4, The significant interaction 
between media selection technique and training objectives precludes any 
interpretation of the superiority of media selection techniques independently 
from the types of learning tasks represented by the training objectives. 

The training system designers also reported the time required to read 
and become familiar with the various techniques. The average time required 
per technique is recorded in table 7. This time varied from 20.0 minutes 
for technique 1 to 84,3 minutes for technique 3 (with the obvious omission 
of technique 0). The mean time for all attempts to learn a technique was 
52.5 minutes. The formal media selection techniques, which ranked highest 
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TABLE 6. ANOVA SUMMARY OF EXPERTS' RATINGS OF MEDIA SELECTIONS 



SOURCE 


df 


MS 


F 


P 


Technique (A) 


9 


19.33 


13.81 


.01 


Training Objective 
(B) 


6 


3.17 


2.26 


.05 


AB Interaction 


54 


4.04 ' 


2.89 


.01 


Within Cells 


770 


1.40 






TOTAL 


839 









TABLE 1. TIME TO LEARN AND USE MEDIA SELECTION TECHNIQUES 



Media 

Selection 

Technique 


Average Time to 
■ Learn System 
(in minutes) 


Ranks 


Average Time per 
Media Selection 
(in minutes) 


Ranks 


0 


0 


1 


11.2 


6 


1 


20.0 


2 


3.5 


1 


2 


50.7 


5 


5.9 


2 


3 


84.3 


10 


33.1 


10 


4 


35.7 


4 


10.4 


4 


5 


62.9 


7 


12.5 


8 


6 


68.6 


8 


17.8 


9 


7 


73.6 


9 


10.9 


5 


8 


25.3 


3 


12.4 


7 


9 


51.4 


6 


9.9 


3 
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in the previous analysis, required 35.7 minutes (technique 4) and 62.9 
minutes (technique 5). 

The time required to make a media choice with the various media 
selection techniques also varied. The mean time to choose a set of media 
ranged from 3.5 minutes (technique 1) to 33.1 (technique 3). The mean time 
for all attempts to make a media choice was 12.8 minutes. The media selection 
techniques, which ranked highest in the previous analysis, required 10.4 
minutes (technique 4), 11.2 minutes (technique 0), and 12.5 minutes 
(technique 5), per media choice. It should be noted that the six training 
system designers were not expert in the use of any of these techniques; the 
data reflects their initial use of these procedures. 

The training system designers also rated each of the 10 techniques 
according to the three scales shown In figure 1 in an attempt to document 
their estimates of the utility of the various techniques. The means of the 
scores assigned by the six designers for each of the 10 techniques is 
presented in figure 3. 

The training system designers ranked the TAEG technique highest on all 
three measures; however, the ratings never exceeded the mid- range on the 
scales. 
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SECTION IV 
CONCLUSIONS AND RECCWMENDATION 
The combined ratings of the two experts for each of the three highest 
rated techniques are essentially similar in value, namely 3.60 (TAEG), 3.55 
(Non-System), and 3»5l (Briggs). Based on these ratings, no case can be 
made for the superiority of any one of these three techniques to be used by 
TAEG in choosing media for proposed training systems. The experimental 
evaluation suggests that the TAEG technique is as good for the TAEG's use as 
any of the other techniques evaluated. Also, none of the techniques were 
found adequate for direct application by the TAEG. Since further develop- 
ment of any of the techniques would be required, the TAEG technique was 
selected as the logical choice for development consonant with the TAEG*s 
needs. 

It Is worth noting that the Non-System technique was rated only slightly 
below the TAEG technique on all measures of usefulness. This traditional 
approach can only be expected to reflect and perpetuate the current state 
of the art. One would expect considerable variation in media choices since 
considerable intuition is demanded of designers in using the i/on-System 
approach. There is also no potential for systematic improvement of this 
technique. • 

RECOMMENDATION - Expand and improve the existing TAEG technique for use in 
TAEG's media selection tasks. Guidelines for this development effort are 
presented in the following paragraphs. Some of these requirements have 
already bcon either partially or fully met by the TAEG technique or other 
existing techniques; other requirements will require significant development 
effort. 
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The required media selection technique should have the following 
characteristics: 

a, .A_ technique for professionals . The media selection technique 
should be designed as a tool to assist professional education technologists 
not to replace th«m. The choosing of instructional methods and media for 
proposed training sy^:^tems remains a professional task that cannot be fully 
proceduralized. In this regard the technique should include the following: 



It should be a formal process with a set of terms operationally 
defined, 

. The logic for media selection should be based upon theoretical 
concepts about the processe.?? of teaching and learning, 

. Human decisions required within the logic should involve variables 
of manageable size and complexity. Adequate information should 
be available to provide a basis for the decisions, 

, The major task of the media selector should be decision making, 
with dat^ retrieval and processing a minor aspect of the task, 
preferably automated. 

, It can be expected that training in the use of the technique will 
be required for skillful use. In turn, the technique should be 
useful in a broad variety of current and emerging training 
situations, 

b. Useful Variables , Variables to be incorporated into the media 



selection logic should be of two types. The first type includes those 
variabl.es that describe a training program as a learning system, i,e,, 
stimulus modes, response modes, feedback modes, learning strategies, and 
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learning styles. The second type includes those variables that are the 
major determiners of the cost of operating a training system^ i.e., ac- 
quisition and operating costs and use rates. 
Specifically the variables include: 

. Task categories with associated learning guidelines. 

. Stimulus characteristics of the tasks. 

. Trainee response modes required by the guidelines. 

. Trainee feedback requirements, 

. Individual learning styles. 

. Economic analysis based on: 
~ life cycle costing 

- use rate projections 

- sensitivity to changing requirements 
Practical factors such as 

- suitability for local production 

- stage of training 

c. Media Alternatives . Media should be selected from an extensive 
pool of alternatives. While none of the tested techniques had more than 
31 media types, approximately 100 generic media types should be in this 
pool. This would cover traditional forms o*f instructional media, new forms 
being introduced into training programs, and theoretical configurations 
of media being discussed in the literature. It would include media for 
supporting classroom instruction and individualized instruction, tele- 
communication, and on-the-job training. Each medium should be carefully 
defined and examples provided. Cost data for the acquisition and use of the 
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media should be included in such a manner that tentative cost comparisons of 
alternative media can be accomplished, 

d. Training System Descriptions . The media selection technique 
should support the preparation of descriptions of proposed training systems. 
Three levels of descriptions are required: 

. Broad instructional strategies for general types of learning 
tasks. 

Specific mixes of media for specific learning tasks. 
Specific design and use parameters in such detail that the 
descriptions can be incorporated into training system 
specifications for the production or procurement of the 
media packages. 

e. Growth Potential . The system should be capable of growth. 
This includes a capability for adding or changing media types, learning 
guidelines, and economic factors. 

f. Interfaces . The media selection technique should interface 
smoothly with both the task analysis formats and training system specifi- 
cation formats to be used in the TAEG. 

g. Data Manipulation . Manual computations and the mechanics of 
combining factors should be a simple task tor the training system designer. 
No more than a few minutes per media selection should be required. Some 
examples of useful techniques include: 

, Two- dimension ^,ables 

, Simple math models 

. Automatic data processing 

. Manual card sort systems 

Q 30 
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Cost Model , Preferably the computing of life cycle costs 
associated with the media selection process should be supported with 
tables of cost factors. The purpose of the cost model is media selection 
not the development of budget estimates. Budget estimates are to be made 
with an in-depth economic analysis. Only a small set of viable training 
system alternatives should be subjected to an economic analysis, 

i. Testing , The media selection technique should be tasted via 
application over a range of the TAEG projects. Appropriate modifications 
and further development should be accomplished resulting in a second gener- 
ation media selection technique that yields optimum results. 
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APPENDIX A 

MEDIA SELECTION TECHNIQUES INCLUDED IN THE EVALUATION 

1. Instructional System Development . AF Manual 50-2, 31 December 1970. 
Department of the Air Force. 

The technique is presented on pages 5-13 and 5-14 under the heading 
"Selection Considerations." It consists of a simple grid type table with 
six types of learning objectives across the top and 10 types of instructional 
media on the left side. Each cell formed by this grid is classified as low, 
medium, or high to indicate the usefulness of the medium in achieving the 
learning objectives. The table was developed by Dr. William H, Allen, 
University of Southern California, and published In the Final Report of 
the Ur»es of Newer Media in Art Education Project , NDEA Project No. 5-16-027, 
National Art Education Association, August 1966. 

2. Bretz, R. The Selection of Appropriate' Communication Media for In - 
struction ; A Guide for Designers of Air Force Technical Training Programs , 
R-601-PR. February 1971. Rand Corporation, Santa Monica, CA, 

Based on a taxonomy of communication media, eight classes of media 
are identified. Thoy are audio-motion-visual, audio- stil 1-visual , audio- 
semimotion, motion-visual, still-visual, semimotion , audio, and print. 
Specific examples of each class are listed! In addition, a series of 
decision flow charts with decision points is presented. By making a series 
of "yes" or "no" ansv;ers to questions presented at these decision points, 
the designer is routed to the required media class. All the media in this 
system are telecommunication or recorded madia. 

3. Siegel, A. I., £< Federman, P. J. Development of a Method for Deriving 
Required Training Aids/Devices and Applications to the Tactical Coordinator 
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Position in ASW Aircraft . Technical Report NAVTRADEVCEN 68-C-0212-1. 
June 1970. Naval Training Device Center, Orlando, Fl. 

In this system, developed by Applied Psychological Services, Inc., 
the designer is required to use a scale, ''Amount of Intellectual Operation,*' 
and rate five intellectual operations in a task or specific behavioral 
objective (SBO). These five operations are cognition, memory, divergent 
production, convergent production, and .evaluation. He is also required to 
use a scale, "Taxation Level of Intellectual Activity," and rate each of 
the five intellectual operations in the task or SBO, These factors are 
combined with predetermined values of the suitability for each instructional 
medium to exercise each of the intellectual operations(values derived from 
the mean of a set of expert opinions). Numerical values that result from 
this procedure are used to determine if the training objective is above the 
difficulty threshold and requires media support. These values are also the 
basis for deriving a rank ordered set of media appropriate for the task. 
Sixteen media are considered. 

4, Staff Study on Cost and Training Effectiveness of Proposed Training 
Systems . TAEG Report T^o. 1. 1972. Naval Training Equipment Center, 
Orlando, FL. 

This report contains a two phase process with initial selection of 
media based on task categories with related learning guidelines and media 
capable of carrying out these guidelines. The second phase involves the 



ERIC 



TAEG REPORT NO. 8 

use of life-cycle costing of a small set of useful media. In the experi- 
mental study being reported, only the first phase was used. In addition, 
a special table was constructed which does not appear in TAEG Report No. 1 
but is based on the 20 page media selection matrix in that report. It con- 
tains 13 task categories along the left side and 20 media alternatives 
across the top. Each cell contained the description **low»" "medium," "high," 
or "not applicable" according to its usefulness in carrying out the learning 
guidelines associated with the task category. 

5# Briggs, L. J. Handbook of Procedures for the Design of Instruction . 
1970. American Institutes for Research, Pittsburgh, PA, p. 93. 

A 14- step procedure is described. Central to this procedure is step 
six which requires the designer to classify the task according to Gagne's 
eight types of learning and then note Gagne^'s special ins -uctional events 
or conditions of learning for the types of learning to be considered. 
Seventeen conditions of learning are identified and related to the eight 
types of learning. From his own experience, the designer is asked to choose 
media that meet the criteria established by the conditions of learning and 
other factors. 

6. Armstrong, G. , et al. The Systems Approach to Training (SAT) Training 
Analysis Guide . August 1971. Air Force Systems Command, Air Force Human 
Resources Laboratory (FT), Williams AFB, Arizona. 

The technique, prepared by Bunker-Ramo, Electronic Systems Division, 
is presented on pages 43 to 62 under headings, "Media Selections" and 
"Application of the Media Selection Technique." It involves classifying 
the tasks according to: (a) three levels of learning, (b) four classes 
of activity, (c) four complexities of cognitive or psycho-motor behavior. 
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(d) three stimulus forms, and (e) three response modes. Five media options 
are available, i.e., the operational system, classroom, learning center, 
simulator, and part-task trainer. Media analysts have compiled tables of 
factors to indicate the usefulness of each medium in supporting each of the 
task factors listed above. Media selection is based on choosing media with 
relatively high overall ratings. 

7. Rhode, W. E. , et al. Analysis and Approach to the Development of a n 
Advanced Multi-media Instructional System . AFHRL-TR.69- 30, Vol. I, May 1970. 
Air Force Human Resources Laboratory, Air Force Systems Command, Wright- 
Patterson Air Force Base, Ohio. 

Although not intended as a media selection technique, this study, 
conducted by the Westinghouse Learning Corporation for AFHRL, contained 
tabulated information including the ranking of media according to cost, 
flexibility, potential for individualized instruction, interaction capacity, 
and cost sensitivity to changes in student load. A simple media selection 
procedure was prepared using these tables. 

8. Walker, R. W. An Evaluation of Tr^lninB Methods and Their Characteris- 
tics. Human Factors ^ 1967, 7, pp. 347- 

A sizeable number of technical training specialists and supervisors 
were asked to list the selection factors they used in deciding which train- 
ing techniques to use. Thirty-four factors were identified. Sixteen 
training techniques were also listed. Then a group of 20 training specialists 
rated each training technique in terms of the selection factors. A Selection 
Criteria Matrix was constructed which contained the mean values for each 
training technique and selection factor pair. 
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To use the matrix, the designer first establishes the real- life 
parameters that must be met in a training program, i.e., short development 
time, low budget, etc. He then examines the matrix to find the training 
technique that best meets the parameters, 

9. Boucher, B. G. , Gottlieb, M. J. h Morganlander , M. L, A Selection 
System and Catalog for Instructional Media and Devices . October 1971. 
Grumman Aerospace Corporation, Bethpage, N. Y. 

This technique was developed in support of the Navy F-14 project. 
Media selection is accomplished through the use of a Media Capabilities 
Matrix. Across the top are 23 media characteristics listed under Presenta- 
tion, Instructional Strategy, and Student Response. Along the left side are 
29 media types. Each cell in the matrix contains one of 11 symbols indicating 
the degree to which the medium contains the media characteristics listed 
across the top of the matrix. To use the matrix, the training system 
designer first determines which of the 23 media characteristics he requires. 
Then by inspecting the matrix he determines which medium best meets these 
requirements. The system is designed for automatic data processing. The 
data processing can also be done by hand. 
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APPENDIX B 
BRIEFING PLAN FOR MEDIA SELECTORS 

One of the reasons the TAEG was established was to develop training 
system specifications. These training specifications incorporate a mix of 
instructional media. The mix might include classrooms, laboratories, 
simulators, and operational systems. The instructional media selected 
should be both adequate to accomplish the training and also be a sound choice 
from the economic analysis point of view. 

The TAEG personnel need one or more techniques for use in selecting 
media for training system specifications. 

Various training system designers have developed techniques for 
determining which instructional media to incorporate into their training 
system designs. They have developed these media selection techniques 
because choosing a training medium can be a perplexing problem. The de- 
signer must choose among hundreds of types of instructional media, each 
suited in varying degrees to various training tasks. 

These different media selection techniques have been described in 
the training literature, but no one technique seems to be widely recognized 
as a superior approach to choosing instructional media, 

Your task is to use some of these media selection techniques--ones 
that appear to be useful to the TAEG operation. You will record media 
selections, and rate each technique in terms of its potential usefulness 
to the TAEG in designing training systems. 

You will use nine different systems and one non- system, or ten 
different methods. A self-contained package of directions, forms, and 
background information has been prepared on each of the methods. 
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Each package contains (1) step- by-step directions for selecting 
media, and (2) background information on the media selection technique. 
You will also be supplied with a set of training objectives and a worksheet. 
Each of these items will now be described in greater detail, 

A set of training objectives is provided which describe? seven spe- 
cific training tasks. This same set of training objectives will be used in 
trying out each of the media selection techniques. Media selected for the 
seven training objectives will be recorded,^ 

The "Worksheet for Media Selectors," figure 4, provides space for a 
media selector to record all his data generated during the trial of a single 
media selection technique. Each media selector, therefore, will fill out 
ten of these forms, one for each technique. 

There are seven items on the worksheet: Item 1 provides space to log 
the number of minutes required to learn the technique prior to its use with 
the seven objectives. If, during the use of the technique, additional back- 
ground study is required, this should be added to the figure originally 
logged in Item 1. 

Item 2 provides space to log the total number of minutes of effort 
required to make media choices for all seven objectives, excluding the 
time logged in Item 1. , 

Item 3 provides space to log the set of useful media alternatives for 
each of the seven training objectives^ Two or more of the most useful 
media options identified through the use of the media selection technique 
should be logged for each objective. 

Items 4, 5, and 6 provide spaces to log a number between 0 and 100 
representing your rating of the media selection technique on three different 
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WORKSHEET FOR MEDIA SELECTORS 



Media Selection Technique: Name: 



1. Time required to learn to use the system: minutes. 

2. Time required to prescribe media for all 7 Training Objectives: 
_minutes. 

3. Media Selections are: 
TRAINING OBJECTIVES: 
(1) 

(2) 
(3) 
(4) 
(5) 
(6) 
(7) 

4. Detail of Media Prescription: 



5. Confidence in Validity of Media Selection Process: 

6. Suitability for TAEG: 

7. Comments: 



Figure 4. Sample of Worksheet Used for Media Selectors. 
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scales. These scales appear on the second page of the Worksheet for Media 
Selectors, 

Item 7 provides space for you to log your comments on the technique-- 
any strong impressions or gripes concerning the logic or lack of logic in 
the approach to media selection and things that should be changed to make 
it more useful. 

Most of the authors of media selection techniques have described their 
processes in detail but have not prepared step- by- step procedures for others 
to use in replicating the method for their own use. Therefore, step-by-step 
directions were prepared for use in this experiment. These directions do 
not go beyond the descriptions prescribed by the originators of the systems. 
If a step appears vague or requires a judgment for which little background 
information has been assembled, it is probably because these qualities were 
in the original descriptions. 

Background information is provided. This is material prepared by 
the original author and will provide the rationale and descriptions of the 
techniques needf^d for you to determine the level of confidence you have in 
the system. Media alternatives are frequently described. You will not be 
able to use a system without first making a careful study of the backRTound 
information ^ although you may choose not to» read every page. Study each 
system until you understand how it works. 

All media selectors will first receive the non- system package. The 
task is to select what you intuitively feel are the best media alternatives 
without the aid of any form of selection technique. Read the training 
objectives and from your experience record the two or more media options 
you believe to be well suited to the training task. 
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To determine the order in which you will use the remaining seven 
packages, select a piece of paper from the box. The piece of paper 
will indicate which of the random sequences you will follow. 

Use one folder each day until you have completed all the folders. 
Pick up a folder each morning between 0800 and 0815 and work on the task 
described in the folder until you have completed the task* Upon finish- 
ing the task, immediately return the folder and worksheet. Work independent- 
ly. Do not receive help from other associates. 

If you have a problem with any procedure and cannot solve it without 
help, discuss the problem with the principal investigator. 

If you must make assumptions concerning the overall training system 
in order to select media for one phase of it, state your assumptions on 
the back of the worksheet. 

Do not discuss the techniques or your media choices with others until 
aftor the experiment has been completed. You will be working on some of 
the packages before other media selectors, and we must not have your work 
influence their media selections. 
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